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Over ihe last two decades, all levels of America's educational enterprise have come under 
increasingly stringent scrutiny. At least since the 1980s, policy makers, business and industr>' 
leaders, and educational reformers have called for increased performance and accountability in 
all aspects of education. While demands for reform are certainly not without precedent, this most 
recent round of criticism is unusual in its protracted length, in the diversity of organizations that 
have issued reports and offered reconrmendations, and in the scope of the reforms that are being 
advocated. 

State governments and accrediting associations have been particularly influential in the 
movement to implement outcomes assessment in postsecondary education. According to Banta 
(1990), almost three-fourths of the states are recommending that their public postsecondary 
institutions make improvements in outcomes assessment. Further, all six of the regional 
accrediting agencies require that their member institutions attend to outcomes assessment issues 
in some way (Marchese. 1 990). Recommendations of these groups have focused on creating 
assessment systems that are sensitive to measuring both educational processes and educational 
outcomes. 

These shifting demands have compelled educators to assess the results of their efforts 
(Ewell, 1989). Postsecondary institutions are increasingly seeking to demonstrate that they arc 
meeting reform requirements by improving educational processes, developing performance 
indicators, and conducting assessment to determine the quality of teaching and learning (Kreider 
& Walleri, 1988), 

Postsecondary vocational education has been affected by each of these reform initiatives. 
It has also been influenced by the Carl D. Perkins Vocational and Applied Technology Education 
Act Amendments of 1990 (informally known as Perkins II). This legislation mandates outcomes 
assessment in vocational curricula at all educational levels. 



Description of the Study 

It is possible to look at both the focus and the methodologies of outcomes assessment 
from either a "traditional" or an "alternative" perspective. In an effort to understand the full 
scope of outcomes practices in postsecondary institutions offering vocational curricula, the 
answers to four research questions were sought. Those questions were: 

1 . What outcomes are currently being measured in postsecondary vocational education? 

2. How are these outcomes being measured by postsecondary institutions? 

3. How are the results used by postsecondary institutions?, and 

4. What changes have occurred at these institutions as a result of these practices? 
Thj*oughout this document, outcomes are viewed as the consequences of involvement in 

or with education. The term "assessment" relates to gathering and analyzing information about 
the impact of education, and about how education can be improved (Sims, 1992). Therefore, 
"outcomes assessment" refers to evaluative processes that determine the results of education. We 
define "traditional" as those outcomes that routinely have been used in the practice of 
postsecondary vocational education, such as job placement, occupational competence, program 
completion or retention, and earnings. These outcomes traditionally have been measured with 
licensure and certification tests, locally-designed tests, follow-up surveys, and self reports. 

"Alternative" outcomes and outcome measures, on the other hand, refer to results that 
have not been used extensively by postsecondary vocational education. Examples of outcomes 
that fall into this category include transfer success, job satisfaction, and individual attainment of 
educational and career goals. "Alternative assessment" approaches can refer to "virtually any 
form of evaluation other than traditional paper and pencil, machine-scored, or multiple-choice 
tests" (Stefonek, 1991, p. 1). The goal of alternative assessment is to produce more valid 
information about student knowledge and skills than has been available from more traditional 
testing procedures. 

There are three types of outcomes to be considered when assessing postsecondary 
vocational education: institutional outcomes, program outcomes, and student outcomes, Within 



the range of educational activities undertaken in postsecondary education, there are certain goals 
that can only be effectively and efficiently addressed at the institutional level. In some cases 
these are outcomes that represent an aggregation of all efforts occurring across the institution, 
such as institution-wide retention rates and student satisfaction. In other cases, the assessment of 
outcomes reflects the result of centralized efforts, such as the impact of economic development 
initiatives or the utility of management information systems. Often, outcomes assessment at this 
level is viewed as part ol institutional effectiveness evaluation (Alfred & Kreiden 1991; Nichols, 
1991; Seybert, 1990). 

Program outcomes, also referred to as functional-area outcomes (Alfred & Kreider, 
1991 ), can serve as a bridg<. between institutional and student outcomes. They provide a means 
by which an institution and its stakeholders can measure the appropriateness and effectiveness of 
specific programmatic efforts. In some cases, programs may be assessed by aggregating student 
outcome measures. Traditional aggregated outcome measures include student enrollment, 
retention, graduation, and job placement rates and employer and graduate satisfaction levels. In 
some cases, however, outcomes must be assessed at the program level because that is where they 
have the most meaning. Often these outcome measures are linked to the efficiency with which 
programs use resources and personnel, and to the effectiveness of services for diverse student 
populations. 

Student outcomes are concerned with changes that occur in individuals as a result of their 
participation in an educational experience. The concept of student outcomes is frequently 
expressed in terms of the value that is added to an individual as a consequence of that 
participation. These outcomes include the knowledge, skill, and attitudes that learners are 
expected to acquire and demonstrate. They are measured by assessing an individual's 
performance. These measures include gains in knowledge and skill competence and are 
frequently measured by competency- or performance-based tests (taking many forms) and 
follow-up surveys. 

During 1991, we conducted research to provide an understanding of the nature of 



outcomes assessment practices carried out by two-year postsecondar>' educational institutions 
and administrative agencies in the United States. In an effort to fully describe the interaction of 
the numerous and diverse factors that come into play in the identification and assessment of 
outcomes, a primarily qualitative research design was chosen for the study. A purposive sample 
of two-year institutions altering vocational curricula was obtained by soliciting nominations 
from experts in the field. These experts included all state community college board presidents, 
all state vocational educali(.)n directors, all American Association of Community Colleges board 
members and selected other officers, and other public and private sector educational authorities 
and leaders. 

The nomination process resulted in the identification of 63 sites located in 25 states 
across the United States. In 9 of the 63 cases (14%), site personnel did not respond to three or 
more attempts to contact them by telephone. Consequently, personnel at 54 institutions were 
interviewed. Table 1 shows the distribution of these sites by accrediting agency. Because the 
geographic size and number of member institutions varies greatly between the regions, these 
findings should not be used to draw inferences regarding the use of outcomes assessment in any 
particular accrediting association. 



Table 1 

Site Distribution bv Accrediting Agency Area 



Accrediting Agency 


Frequency 


Per Cent 


Middle States 


8 


15% 


New England 


2 


4% 


North Central 


23 


43% 


Northwest 


7 


13% 


Southern 


12 


22% 


Western 


2 


4% 



Demographic information on the sites, including total non-duplicated enrollment, the 
proportion of students classified as full-time, and the proportion of students enrolled in 
vocational or occupational curricula, was collected during the inlen'iew. The data is from Fall 
Semester. 1990. The colleges ranged in size from 712 to 92,000 students enrolled in college 
credit courses. The ration of part-time to full-time students and career program to non-career 
program students also varied widely (Table 2). 

Table 2 

Fall 1990 Characteristics of Interviewed Institutions 

Mean Range Standard 

Deviation 

Non-Duplicated Enrollment 15,757 712 - 92,000 20,211.2 

Per Cent Enrolled Full 40.2% 4- 100% 24.1% 
Time 

Per Cent in Vocational 51.8% 10.9 - 100% 24.5% 
Curricula 



The sample obtained through the nomination process was not intended to be generalizable 
to other institutions. Rather, the intent was to determine what practices are being used at 
institutions identified by persons knowledgeable in this area. The underlying assumption was that 
examining only the nominated sites would increase the chances of uncovering a wide range of 
practices while keeping the sample size at a reasonable level. Data was collected from the 
nominated sites using a field-tested, semi-structured telephone interview. Responses from all 
interviews were recorded on a standard response sheet. In all cases, interviewees granted 
permission to record the interviews with the understanding that the recordings would remain 
confidential. 
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Data analysis was based on content analysis of the responses. Two analysts, working 
independently, reviewed the written records of the interviews and created frequency distributions 
of the responses to the various questions. The two lists were then compared. Where there were 
diiVerences, the written records were examined to resolve the differences. The tape recordings 
were also used as needed to clarify the notes that had been made on the response sheet. 

Findings 

At the beginning of the telephone interview, respondents were asked to describe what 
their institution is doing in the arec of outcomes identification and assessment. The responses 
provided a profile of how persomiel at each site are implementing outcomes assessment 
initiatives with respect to vocational education. Most (78%) of the institutions reported that ihey 
are conducting outcomes identification and assessment processes as part of an institution-wide 
ettort. Of the twelve sites that reported that their efforts applied only to vocational education 
students, four were exclusively vocational instruction facilities. At 84% of the comprehensive 
education institutions, then, the outcomes identification and assessment practice includes both 
vocational and non-vocational curricula. 

Responses also showed that 59% of the institutions surveyed believe that at least some 
part of their outcomes identification and evaluation process has been fully implemented. The 
next most frequent response, indicating that development efforts are underway, was given by 
33% of the respondents. The middle ground, field testing and/or revising, was reported by only 
24% of the institutions. Although the issue was not pursued during the interview, the interviews 
and the field data both indicate that field testing and revision prior to full implementation are not 
priorities at many colleges; once a practice has been developed, it is implemented across the 
board. The distribution of the responses is shown in Table 3. 

Readers will note that some tables reporting survey data indicate more than 54 responses. 
This reflects the fact that respondents gave multiple answers to some questions. For example, 
they may have stated that some parts of their outcomes assessment process are fully 
implemented, while other parts are under development. In these cases, the reported percentages 
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Table 3 

Reported Level of Outcomes Program Maturity 



Status Frequency Per Cent 

Developing 18 33% 
Field Testing 7 13% 
Revising 6 11% 
Fully Implementing 32 59% 



also exceed 100%. 

Early in the interview, respondents were asked what types of alternative outcomes were 
being measured at their site. In order to capture the respondents' sense of what constitutes an 
alternative outcome and thereby gain a broad perspective on how practitioners operationalize the 
concept, the term ''alternative" was not defined. Several colleges indicated that they were not 
doing anything "alternative'' with regards to outcomes assessment. The most frequently given 
responses were student academic achievement and student success upon transfer to another 
educational institution (typically a four-year college or university); each was cited by 24% of the 
respondents. Student satisfaction was mentioned in 21% of the interviews. Given that the 
interviewees were specifically discussing outcomes of vocational/technical education at their 
institution, these responses seem to indicate that educational performance and student and 
employer satisfaction with the services provided are seen as important outcomes for 
postsecondary vocational education. The most frequent responses are shown in Table 4. 

During the telephone interview, respondents frequently described the processes they use 
to measure the extent to which desired outcome occur. Recently, the use of standardized tests, 
projects, and portfolios to measure academic achievement has received considerable attention. 
According to the interviewees, these techniques are being used in postsecondary vocational 
education. Interviewees reported that surveys are frequently used to measure a wide range of 
outcomes, including academic achievement, student and employer satisfaction, job placement. 
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Table 4 

Partial Frequency Distribution of Alternativg Outcgmgs Bgins Mgasmiad 



Identified Outcome 


Frequency 


Per Cent 


Academic Achievement 


13 


24% 


Transfer Success 


13 


24% 


Student Satisfaction 


i 1 


20% 


Employer Satisfaction 


10 


19% 


Attainment of Student Goals 


7 


13% 


Employment 


6 


11% 


Licensure Exam Pass Rate 


6 


11% 


Skill Attairunent 


6 


11% 


Job Placement 


4 


7% 


Student Persistence 


4 


7% 



goal attainment, and transfer success. Measurement of transfer success has also been enhanced 
by the implementation of data-sharing mechanisms that link two-year and four-year institutions. 
The measures identified for the most commonly mentioned outcomes (from Table 4) are shown 
in Table 5. 

Throughout the interviews, the identity of primary stakeholders and the origin(s) of 
driving force(s) that led institutions to initiate or modify their outcomes practices were sought. 
Respondents indicated a strong perception that their practices are camyus-based. In 63% of the 
cases, both college administrators and college faculty were i^^^^ntllied as stakeholders; students 
were mentioned in 31% of the interviews. Business/industry, the third most frequently given 
response (39%), was the only off-campus group to be widely recognized. (See Table 6). 

Site administrators were the most frequently mentioned driving force behind changes in 
outcomes practices; they were mentioned in 35% of the interviews. Mandates from accrediting 
agencies (22%) and state government (20%) were also mentioned fairly often. Business/industry 
groups, who were frequently identified as stakeholders, were not specifically mentioned in this 
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Table 5 

Reported Moans By Which Identified Outcomes Are Measured 



Identified Outcome Identified Measure(s) 



Academic Achievement 


1 m • t A A A A 1 1 t A 1 t * t A 

Standardized tests, state-developed standardized tests. 




DnrtfolifiQ evit te^tQ t^rnHincr of work Kv a njinel of fnriiltv 




faculty surveys, student surveys, capstone projects, capstone 




tests 


Attainment of Student Goals 


student surveys, tracking system 


Employer Satisfaction 


employer surveys 


Employment 


state unemployment records, college placement office data 


Job Placement 


surveys 


Licensure Exam Pass Rate 


test scores, pass-fail rate 


Skill Attainment 


on-site assessment, national competence exams, licensure 




exam scores, state and national competitions, narrative 




assessment, satisfactory/unsatisfactory performance rating, 




internships 


Student Persistence 


enrollment patterns 


Student Satisfaction 


student surveys 


1 ransfer Success 


transcripts, state tracking system, grade reports, success at 




receiving institution, feedback from receiving institutions 



regard. Community needs and trade labor unions were each mentioned only once. Further, 
faculty were not identified as a driving force for change by any institution. Table 7 shows the 
full set of responses. 

During the telephone interview, respondents were also asked to discuss how the results of 
their institution's outcomes process are used. Respondents stated that institutions mainly use the 
information gained from outcomes assessments for program evaluation (33%), strategic planning 
(31 %), and curriculum development purposes (26%), Outcomes data was also frequently cited as 
a factor in determining how resources are allocated (19%). 
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Table 6 

Idgniilkd 3takghold<;r^ in OuKomgs Practices 



Identified Stakeholders 


Freq uency 


rQT cent 


College Administrators 


34 


63% 


College Faculty 


34 


63% 


Business/Industry 


21 


39% 


College Students 


17 


32% 


State Policy 


2 


4% 


Auxiliary Personnel 


1 


2% 


College Trustees 


1 


2% 


Community Groups 


1 


2% 


Labor Unions 


1 


2% 



Table 7 

Driving Force Behind Changes in Outcomes Practices 



Impetus 


Frequency 


Per Cent 


Site Administrators 


19 


35% 


Accrediting Board Mandate 


12 


22% 


State Mandate 


11 


20% 


Process Evolution 


9 


17% 


Federal Mandate 


6 


11% 


Internal Study 


4 


7% 


Self Study 


2 


4% 


State Grant 


2 


4% 


Board of Trustees 




2% 


Community Needs 




2% 


Competition with Other Institutions 




2% 


Institutional Research Office 




2% 


Labor Unions 




2% 



^1! 
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Only 1 1% of the respondenls indicated thai the outcomes data gathered is used for 
accountability reporting. This might indicate that although outcomes identification and 
measurement is often mentioned as a reform directed at improving accountability, most 
practitioners do not find accountability reporting to be an important consequence of their 
outcomes programs. Further, while student outcomes dominated the list of outcomes that were 
identified as important by practitioners (Table 4), only 7% of the respondents said that outcomes 
data are used to assess students. This may suggest that the types of practitioners inter\aewed. 
mainly institutional researchers, do not typically view student outcomes and student assessment 
as interrelated practices. Table 8 provides a frequency distribution of responses. 

Table 8 

Uses of Outcomes Data 



Use of Data 


Frequency 


Per Cent 


Program Evaluation 


18 


33% 


Strategic Planning 


17 


31% 


(.'urriculum Development 


14 


26% 


Resource Allocation 


10 


19% 


Progrun) l^lanning 


9 


17% 


Accountability Reporting 


6 


11% 


I'ublic Relations 


6 


11% 


Student Assessment 


4 


7% 


School - Husiness/lndustr>' Coordination 


3 


6% 


Student Services Improvement 


3 


6% 


Denuigraphic Trend Monitoring 


2 


4% 


Instituti .ul lilTcctivcness Improvement 


2 


4% 
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During the telephone survey, respondents were asked whether "outcomes identification 
and evaluation has changed the college in any way?" In 89% of the interviews, respondents said 
that there had been changes as a result of outcomes assessment. At 8% of the institutions, 
interviewees believed that changes were beginning to occur as a result of these efforts. The 
remaining interviewees did not see that there had been any change. 

At those institutions where changes were believed to have occurred, or to be occurring, 
better decision making and planning was seen as the primary improvement (30%). Improved 
instruction (29%) and improved accountability (23%) were also frequently cited. These findings 
agree with the fmdings reported in Table 6, where issues related to planning, resource allocation, 
accountability, public relations, and curriculum development were commonly cited uses of 
outcomes data. From these two sets of responses, it seems possible to infer that many of the 
changes that are occurring are the result of the utilization of data collected through the outcomes 
process. The full list of responses is reported in Table 9. 

Summary and Conclusions 

This study suggests several major findings. The first is that the outcomes currently being 
measured in postsecondary vocational education generally seem to correspond to the changes 
sought by policy makers and education reformers. The outcomes most frequently identified by 
interviewees (academic success, transfer success, student satisfaction, employer satisfaction, 
attainment of student goals, employment, licensure exam pass rate, skill attainment, job 
placement and student persistence) all represent an expansion of the focus of postsecondary 
vocational education. However, the diversity of outcomes identified, and the relatively low 
frequency with which most outcomes were reported, suggests that no consensus on outcomes has 
formed in postsecondary vocational education. 

The emphasis on student goal attainment has the potential to be problematic for some 
institutions. As long as institutional outcomes are assessed by performance against such 
traditional standards as program completion and graduation rate, the tension between serving 
students who only wish to take one or two classes (or, perhaps, only parts of one or two classes) 
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Table 9 

Changes See n Resulting I'rom Outcomes Practices 



Change 


Frequency 


Per Cent 


Belter Decision Making and Planning 


16 


30% 


Improved Instruction 


15 


28% 


Improved Accountability 


12 


22% 


Better Staff Development 


4 


7% 


Faculty See and Use Results 


4 


7% 


Institution is Better Attuned to Business Needs 


4 


7% 


Climate Supports On-going Assessment 


b 


Dyo 


More Confident of Institutional Effectiveness 


3 


6% 


Decision Making Has a Feedback Loop 


2 


4% 


Faculty More Conscious of Outcomes 


2 


4% 


Helps Everyone See the "Big Picture" 


1 


2% 


Higher Expectations of Faculty and Students 


1 


2% 


Student Perceptions Incorporated in Program Changes 


1 


2% 



and demonstrating programmatic and institutional effectiveness based on degree completion will 
be a source of aggravation for administrators and confusion for legislatures and the public. 
Efforts are needed to redefine and reorganize effectiveness and funding formulae to recognize the 
importance of helping students meet their specific educational goals regardless of whether those 
goals include degree or certificate completion. 

A second finding is that outcomes measurement practices are changing rapidly as 
instructors seek to incorporate new techniques into their instructional programs. Portfolios are 
being introduced in a variety of curricula, as are capstone projects, capstone tests, and other 
means of performance assessment. This is, most likely, a positive development so long as the 
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faculty using these instruments have the training needed to properly design and use them. As one 
institution indicated, this is not always the case. There, efforts are underway to develop in many 
instnictors sufficient psychometric skills to facilitate the successful use of practices that have 
already been implemented. The implementation of new assessment practices might have been 
much less stressful had administrators provided adequate in-service training prior to launching 
the initiative. As institutions seek to expand their assessment techniques, the need for staff 
development in these areas will most likely increase. Unexpected difficulties presented by 
expanding into new areas indicate that improved feedback and process adjustment mechanisms, 
including opportunities for field testing, are needed between development and full 
implementation. 

A third finding is that site administrators are seen as being both stakeholders in and 
driving forces behind changes in outcomes practices. No other group was highly ranked in both 
categories. Faculty, business and industry, and, to a lesser extent, students £ire also seen as being 
stakeholders. The other major driving forces were accrediting boards and state government. 

Finally, because the development of an outcomes program is typically a time consuming 
and stressful process, it is important to kjiow whether institutions and individuals can reasonably 
expect to receive benefits that justify their efforts. The institutions interviewed stated that 
outcomes data has improved their ability to address internal issues. Chief among these issues 
were program evaluation and strategic planning, Curriculun'i development, resource allocation, 
and program planning were also mentioned. As mentioned earlier, despite the emphasis on 
student performance assessment that was indicated in Table 4, only a small number of 
interviewees mentioned student assessment as a use of the data. This may reflect a bias of the 
interviewees (mainly institutional researchers) toward institutional rather than student outcomes. 

Some of the changes that were mentioned less frequently seem to be among the more 
interesting that were reported. The need to incorporate a feedback loop into decision-making 
processes has been previously discussed; two respondents indicated that their outcomes practices 
provide such a mechanism. Other institutions might be well advised to study ways that they can 



incorporate provisions for providing opportunities for feedback into their processes. Doing so 
might contribute to a more general reaHzation of two more infrequently mentioned changes: 
development of a climate that supports on-going assessment, and improvement of the ability of 
faculty to see the "big picture" of educational practices at their institution. 

One site indicated that its outcomes practices have raised the expectations of faculty and 
students. If this is true, then the intent of many reforms has been realized at that college. The 
prospect that outcomes identification and measurement can bring about such a change is a 
powerful argument in favor of its continued prominence as a mechanism for effecting reform. 

Recommendations for Further Study 

Tliis study in no way represents the end of the investigation of outcomes identification 
and assessment issues in postsecondar\' vocational education. Indeed, it is just the beginning; the 
study only describes what selected individuals at a non-representative sample of institutions 
reported that their colleges are doing. This does not minimize the importance of the research, as 
it appears to be the first national survey of practice. It does, however emphasize that much 
remains to be done. 

Perhaps the next step should be to determine the generaiizability of efforts at sites thai 
were studied to other institutions with similar goals. What factors would enhance the 
transferability of outcomes approaches? Which factors would inhibit it? W ith this information 
in hand, it would be possible to encourage the sharing of ideas among institutions with similar 
needs, thereby increasing the number of talented faculty and administrators working to resolve 
these complex issues. An expansion of the knowledge base concerning outcomes identification 
and assessment would surely enhance efforts to bring about positive change in edueatiunal 
practice. 
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